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Abstract — We treat a random number generation from an i.i.d. 
probability distribution of P to that of Q. When Q or P is 
a uniform distribution, the problems have been well-known as 
the uniform random number generation and the resolvability 
problem respectively, and analyzed not only in the context of 
the first order asymptotic theory but also that in the second 
asymptotic theory. On the other hand, when both P and Q 
are not a uniform distribution, the second order asymptotics 
has not been treated. In this paper, we focus on the second 
order asymptotics of a random number generation for arbitrary 
probability distributions P and Q on a finite set. In particular, 
we derive the optimal second order generation rate under an 
arbitrary permissible confidence coefficient. 

I. Introduction 

The random number generation is one of the most basic 
problems in the information theory. The purpose of the random 
number generation is to approximate a sequence of target 
probability distributions Q n by transforming a sequence of 
another probability distributions P n . When Q n or P„ is a 
uniform distribution, each problem corresponds to the uni- 
form random number generation problem or the resolvability 
problem respectively, and has been well studied. For example, 
when P n and Q n are the i.i.d. probability distributions P" 
and Ug 71 where U2 is the uniform distribution with the support 
size 2, the optimal first order generation rate a from P n is the 
entropy H(P) under the condition that the error goes to 0. 
Those problems are analyzed not only in the context of the first 
order asymptotic theory but also that in the second asymptotic 
theory. In the most general case, it is known that the first 
and the second order optimal rates in those problems can be 
described by the information spectrum methods [fl], El, 0. 
In particular, for transformation between P n and jj^ n+b ^^ 
the results in the information spectrum gives optimal rates 
a and b. On the other hand, when both P n and Q n are not 
a uniform, the problem has not been treated sufficiently. In 
this paper, we do not restrict both P n and Q n to a uniform 
distribution and focus on the second order asymptotics of 
a random number generation for arbitrary i.i.d. probability 
distributions on a finite set. In particular, we derive the optimal 
second order generation rate under an arbitrary permissible 
confident coefficient. 

In this paper, we utilize the notion of the majorization. It 
is a pre-order between two probability distributions which can 



be defined on different finite sets. If a probability distribution 
P n is transformed to W n (P n ) by a deterministic transfor- 
mation W n , the transformed probability distribution W n (P n ) 
"majorizes" the original probability distribution P n . In other 
words, W n (P n ) is larger than P n in the sense of the ma- 
jorization relation. Therefore, when we want to approximate a 
target probability distribution Q„ from an original probability 
distribution P n , for an arbitrary deterministic transformation 
W n , there is a probability distribution P' n which majorizes P n 
and is close to Q n than W n (P n ). Thus, the performance of the 
optimization under the majorization condition gives a bound 
of that under deterministic transformations. The majorization 
is used in a transformation theory of quantum entangled states 
and corresponds to a operation called LOCC in the quantum 
information theory Q, J8). Our results can be extended to the 
quantum settings but we do not mention it in this paper. 

The paper is organized as follows. In section II, we in- 
troduce a notion of majorization, and consider approximation 
problems under a majorization condition and by a determin- 
istic transformation. In section III, we treat the first order 
asymptotics for the approximation problem and derive the first 
order optimal rate under the i.i.d. setting. In section IV, we 
review the existing studies about the second order asymptotics 
of the approximation problem when the source distribution or 
the target distribution is a uniform distribution. In section V, 
we treat the second order asymptotics of the approximation 
problem when both the source distribution and the target 
distribution are not a uniform distribution. We note that the 
results itself in section V do not contain that in section IV. 
But, in the end of section V, we show that the result in Section 
IV can be regarded as the limit case of the sesult in section 
V. In section VI, we state the conclusion of the paper. 

II. One-Shot Formulation 



In this section, we introduce some notation and definition, 
and formulate our problem. For a probability distribution 
P on finite set X and a map W : X — > y, the proba- 
bility distribution W(P) on y is defined by W(P){y) := 
YlxeW-ifx') ^ > ( x )- ^ e introduce a value F called the Bhat- 
tacharyya coefficient or the fidelity between probability distri- 



butions over the same discrete set y as 



F(Q,Q') := vW \fW)- 
yey 



(1) 



This value F represents how close two probability distri- 
butions are and relates to the Hellinger distance dn as 
dn(', ■) = yl — F(-, ■). Then our main purpose is to analyze 
the following value. 

L D (P, Q\v) := mnx{L\F(W(P),Q L ) > v, W : X -> y L }(2) 

This means the maximal number L of C] L which can be ap- 
proximated from P under a confidence coefficient < v < 1. 
When we define the maximal fidelity F D from P on X to Q 
on y by 

F B (P -> Q) := m&x{F(W(P), Q)\W : X — > y}, (3) 
Then L D is rewritten as 



L D (P, Q|i/) = max{L\F D (P -> Q L ) > i/}. 



(4) 



Next, we will introduce the notion of the majorization to 
evaluate F D . For a probability distribution P on a finite set, 
let P+ be a sequence {P/}^ where P^ is the element of 
{P(a;)} xe ,* sorted in decreasing order for 1 < i < \X\ and 
Pf is for \X\ < i. We set as C P (l) := J2\=i p t- when 
probability distributions P and Q satisfy C P i(l) < Cqi(1) 
for any I £ N, it is said that P is majorized by Q and written 
as P -< Q. Here, note that the sets where P and Q are defined 
do not necessarily coincide with each other. The majorization 
relation is a pre-order on a set of probability distributions in 
which each distribution is defined on a finite set[6|. We intro- 
duce the maximal fidelity under the majorization condition. 



P M (P -> Q) := max{P(P', Q)\P -< P' on y} 



(5) 



where P and Q are probability distribution on X and y, 
respectively. Since P -< W(P) for a map W : X — > 3^, 

F M (P^Q) > F D (P 



Q) 



(6) 



holds. In many case, P M is easily treatable than P D . In 
particular, the value of F M can be explicitly described in one- 
shot situation Q. 

III. First Order Asymptotics 

We proceed to the asymptotics of random number genera- 
tion. Here, we assume the form P n and Q an as P and Q in 
the last section. Then, we call a > the first order rate. In this 
section, we derive the dependency of F D (P n — > Q an ) for the 
first order rate a when n goes to the infinity. The following 
theorem say that the threshold value of the first order rate is 
the ratio of the entropy of P and Q. 

Theorem 1: 

limF D (P n -+Q an ) = limP M (P" -> Q an ) 



if a < 
if a > 



H(P) 
H(Q) 
H(P) 
H(Q) ' 



(7) 



The limit of the maximal fidelity is obtained by Theo- 
rem [TJ when the first order rate is not H(P)/H(Q). We 
rewrite L D (P n , Q\v) as L^(P, Q\v). Then, we can not obtain 
even the first order asymptotic expansion of L^(P, Q\v) 
for < v < 1 from Theorem Q] because the case then 
a = H(P)/H(Q) is not treated in TheoremQ] In other words, 
we can not derive the performance of LP (P,Q\v) under a 
confidence coefficient < v < 1 when n is large. Therefore, 
we fix the first order rate a to H{P)/H(Q) and analyze the 
second order rate in the next section. 

IV. Second Order Asymptotics for Uniform 
Distribution 

We focus on the approximation from P™ to Q an + b Vn anc j 
assume that the first order rate a is H(P)/H(Q). We treat the 
following problems. 

(I) For a fixed second order asymptotic rate b 6 R of 
QH(p)/H(Q)n+b^Ti^ we derive the limit value of the maximal 
fidelity lim F D {P n -> QH{P)/H{Q)n+b^y 

(II) For a fixed confidence coefficient < v < 1, we derive 
the second order asymptotic expansion of L^(P, Q\v). 
These problems are essentially equivalent. To derive the sec- 
ond order asymptotic expansion under a confidence coefficient 
< v < 1, we focus on the following value which means the 
optimal second order rate. 



sup I b G R \imF D {P n -> Q 



H(F) 
H(Q) ' 



> V 



}■ (8) 
asymptotic 



If the above value is finite, the second order 
expansion of L^(P,Q\v) is represented as 

= (H(P)/H(Q))n + R°(P,Q\v)Vn~ + o(VTi). (9) 

We review the results for the uniform random number 
generation and the resolvability in 0. Let Ui be the 
uniform distribution with the support size 2. Then the limit 
value of the maximal fidelity is represented as follows for a 
fixed second order rate b £ R. 

Theorem 2: Let P be any probability distribution on a finite 
set except for a uniform distribution. Then 



lim F D (P r ' 



= lim F M (P 



U. 



\ 



1 -G 



where H(P) is the entropy of P and 

V(P) := ]T P(x)(-logP(x) - H(P)) 2 . (10) 

From Theorem [2] it turned out that the limit of the 
maximal fidelity depend on the second order rate b when 
a = H(P)/H(Q). Note that the limit value does not depend 
on the second order rate b when a ^ H{P)/H{Q) is given 
by Theorem [TJ We emphasize that the lower order term does 



not affect the limit value if an + by/n has lower order term 
as an + by/n + o(y/n) (e.g. o(y/n) = logn). Hence, when we 
want to analyze the maximum fidelity, we only have to treat 
the first and second order rate and do not need the third order 
asymptotics. For a fixed confidence coefficient < v < 1, the 
second order asymptotic expansion is represented as follows. 

Theorem 3: (3) Let P be any probability distribution on a 
finite set except for a uniform distribution. Then the second 
order asymptotic expansions in (O are described as follows. 

L»{P,U 2 \v) 



H(P)n- y /V(P)G- 1 (v 2 )Vn~+o(V^), (H) 



del. Theorems which appear later are represented by those 
symbols. 

N P := N(0,V(P)), (15) 
Np,Q,b:=N\H(Q)b,^^V(Q)Y (16) 



G P {x) := G 



G 



P,Q,b 



x) := G 



H(Q) 



(17) 



W(Q) (I ^H' (18) 



where G is the cumulative distribution function of the standard 
normal distribution. Here, 



B§{P,U 2 \v) 



Next we consider the approximation from the uniform 
distribution to P. The limit value of the maximal fidelity is 
represented as follows for a fixed second order rate tel. 

Theorem 4: Let P be any probability distribution on a finite 
set except for a uniform distribution. Then 



\imF D {U y 2 



pH{P)- 1 n+b^Ti 

]imF M (U$^P H ^~ lr 



G 



-h(p)H 



(12) 
(13) 



For a fixed confidence coefficient < v < 1, the second order 
asymptotic expansion is represented as follows. 

Theorem 5: (2) Let P be any probability distribution on a 
finite set except for a uniform distribution. Then the second 
order asymptotic expansions in (0 are described as follows. 

L°(U 3 ,P\u) 



H{Py 1 n 



' V(P) 
H{P) 3 



G- 1 (^ 2 )V^ + o(V^). (14) 



Here, 



RV(U 2 ,P\v) = -J^G-H^ 



V. Second Order Asymptotics for Non Uniform 
Distribution 



In this section, we treat non-uniform distribution cases. We 
note that the results itself in this section do not contain that 
in the section IV because we use the property that both V(P) 
and V(Q) are not 0, which is equivalent to that both P and 
Q are not uniform distributions. But, as is shown in the end 
of this section, the result in section IV can be regarded as the 
limit case of the result in section V. We define some notations 
for non-uniform probability distributions P, Q and a constant 



/ 2JC PO < g w» b » 2 

Ip,Q,b{x) :=\ L W I' Q e ^cV )(! +o>, Q) 



1 + C P , 



x G 



1 + C P , 



2V(P)C P ,Q V 1 + C P . 



11.(19, 



[ P,Q,b 



(oo) := J V e 4v(P)(, + c PiQ) 

y 1 + Cp.q 



(20) 



where N(fi, v) is the normal distribution with the mean fj, 

and the variance v, and Cp.q :— v(p) ( v(Q) ) ■ Note that 
Gp,Gp,Q,b means the cumulative distribution functions of 

Np,N P ,Qj and 



Ip,Q,b{x)= / y/N P {t)yjN PtQtX (t)dt, (21) 
I P ,Q,b{™)= I V N p(t)^N P ,Q, x (t)dt (22) 



hold. 

H(P) , r- 

We consider the approximation from P n to <3 H(Q) . 
There are three cases for the limit value of the maximal fidelity 
by the relation of y[pj and • The following is the first 
case. 

Theorem 6: When y|pj > v(Q) ' 

N P (x) G P (x) 



(23) 

Np,Q, b (x) Gpq :6 (x) 

has the unique solution a £ K with respect to x, and the 
following holds. 

YimF D {P n -> Q^Sj n+b ^) 
= limF M (P" -> QmB n+b ^) (24) 



= ^Gp{a)^l Gp tQtb (a) + I PtQtb (oo) - Ip,Q,b(a) (25) 
=: Fi(b) (26) 
For the continuous differentiable function 



Gp(a) 



Mix) = \ Gf.Q.\ 

[ Gp{x) 
the following equation holds 



^Gp.q.bix) \ix<a 
if a < x, 



(28) 




Fig. 1. Let H(P) /V(P) > H{Q) /V{Q). The red, the blue and the orange 
lines show Gp, Gp g^ and c ^ p ^ n j Cp,Q,&. respectively. Then, A\ is 
represented as the orange line oni^a and the red line on a < x. The limit 
of the maximal fidelity in Theorem [6] coincides with the fidelity between A± 
and the blue line Gp q },. 



where F (jr",Np t Q t b) is the value defined as follows and 
is called the fidelity or the Bhattacharyya coefficient for 
continuous distributions. 



F(p,q):= / yjp{x)yjq{x)dx. 



(29) 



Therefore Theorem [6] can be represented as 



YimF D (P n -> Q- 



limF M (P 



The positional relation of the functions Gp,Gp,Q,b and A\ 
is shown in Fig[T] This fact holds for the following theorems 
about the limit of the maximal fidelity. In other words, the limit 
of the maximal fidelity can be represented by a continuous 
differentiable distribution function A on R as follows. 



WmF u {P n -> QTm n+b ^) 



\xmF M (P n -> Q~t 



dA 



Vb ^) = F[ =jl,N P , Q , b )pi) 



where a distribution function on R is defined as a right- 
continuous increasing function which satisfies lim A(x) = 

X— OO 

and lim A(x) = 1. 

x— >oo 

To prove Theorem |6] we give the sketch of proof of (1301 . 
First, the following lemma is essential for the proof of the 
direct part. 

Lemma 1: Let P and Q be probability distributions. When 
a function A on R is continuously differentiable, monotone 
increasing and Gp < A < 1, the following holds. 



liminfF D ( P n -> Q 



dA 

>F[-£,N P ,Q, b ). (32) 



We do not provide the proof of the above lemma here. By this 
lemma for A — A\, the left term in ( f30b is greater than or 



equal to the right term. Secondly, taking the limit superior in 
©, we obtain 



limsup J F ^5 (P ,l -> Q#Sl n +^) 
< limsupF M (P" -> Q#Sl"+ 6 ^). 



(33) 



Therefore, the middle term in (l3(Tt is greater than or equal to 
the left term. Lastly, we will explain that the right term in ([30i > 
is greater than or equal to the middle term. We introduce the 
notation of subset in N as A n {x) := {1, 2, • • •, [ e H(P)n+x^^ 
and _4„(x,x') := \ A n (x). Then, for an arbitrary 

< e and probability distributions {Pn}^=i sucn that Pn 
is majorized by each P' n , let us take a real number c € R 
which satisfies 



y/l-G P (c)Jl - G P , Qtb (c) < e. 



(34) 



and a < c where a is the unique solution in ( f23l . Moreover, 
for a natural number /, let x\ be a + ^j^i. Since the 
monotonicity of the fidelity, 

F(P',Qn) 



< ^P^A n {x ))^Q n {A n (x )) 
i 



(35) 



+ y/PJ l (A n (c, Oo))y/Q n (An(c, Oo)) 

By direct calculation, when n goes to oo 

lirnsupF(P" ->■ Q$^ n + 6 ^) 

< y/Gp(a)y/G P ,Q, b (a) + / y/N P (x)y/N PiQib (x)dx (36) 

+V1-Gp(c)^1-Gp,q, 6 (c) 

< ^OpJ^^^Gp^ia) + f x/Np(x)^N P:Q:b (x)dx + e 

<37) 

holds. Therefore, the right term in ((30) is greater than or equal 
to the middle term. Taken together, we obtain (l30l from (l32t . 
433) and (|37). 

Theorem 7: When v(P) > v (q) ' tne seconc ^ or der asymp- 
totic expansion for a confidence coefficient < v < 1 in (0 
is described as follows. 

L D (P,Q\v) = (H(P)/H(Q))n-F 1 - 1 (^, + o(^i). 

In particular, R 2 {P,Q\v) = R§(P,Q\v) = F^{v). 

The following treat the second case on the limit value of 
the maximal fidelity. 

Theorem 8: When #^ < 



v{P) ^ y(Q)' 
N P (x) _ 1 - G p (.t) 



Np,Q,b(x) 1 - G P ,Qfi{x) 



(38) 



Fig. 2. Let H(P) /V(P) < H{Q) /V{Q). The red, the blue and the orange 
lines show Gp, GpQ^ and 1— i^^~^^TWj (l-Gp,Q,tH), respectively. 
Then, A2 is represented as the red line on x < f3 and the orange line on 
ft < x. The limit of the maximal fidelity in Theorem [8] coincides with the 
fidelity between A2 and the blue line Gp g j. 



has the unique solution f3 S 
following holds. 



max, 
> ff(p) 1 



with respect to a;, and the 



H(Q) , /— 



limF^P" 
= limF^ (P r ' 
= Ip.Q.bifi) + ^l-G P {f3)^l-Gp, Q , b {fi) 
=■ F 2 (b) (39) 

Taking the following function as a function A : K — > [0,1], 
Theorem [8] can be represented as ( |3~TV 



4«(x) = 



Gp(i) 



1 - 



I-Gp(^) 



l-G, 



if x < (3 

(l-G P , Q , b (x)) iif3<x. 



The positional relation of Gp,Gp^Q^ and A2 is shown in 
FiglD Theorem [8] can be proven as the same as Theorem [6] 
by the combination of (l32l i. ( l3"3T l and (f3Tb . But, we need to 
correct the threshold points a and c in (l37l i and change the 
area where the function \J Np^jNp : Q :X is integrated. 

Theorem 9: When yj-pj < v(q] ' tne secon d order asymp- 
totic expansions for a confidence coefficient < ^ < 1 in (0 
are described as follows. 

L%(P,Q\u) = (H(P)/H(Q))n-F 2 - 1 ( I y)^i + o(^i). 

In particular, R 2 (P,Q\v) = R^{P,Q\v) = F 2 ~ 1 (v). 

The following treat the third case on the limit value of the 
maximal fidelity. 

Theorem 10: When ^|p| = ^77757, the following holds. 



V(Q)- 



\imF D (P n — S 
= limF M (P" 



H(Q) , i— 

> QHW) n+b v") = 



e sv, ( p ) if b < 



1 



if 6> 



Taking the cumulative distribution function Gp as a function 
A : M — » [0, 1] Theorem [TOfcan be represented as (|3H . The 



Fig. 3. Let H(P)/V{P) = H(Q)/V(Q). The red and blue lines show G P 
and GpQ 1, respectively. The limit of the maximal fidelity in Theorem 1101 
coincides with the fidelity between the red line Gp and the blue line Gp q b 



positional relation of Gp, Gp t Q t b is shown in Fig[3] Theorem 
[H)]can be proven as the same as Theorem|6]by the combination 
of d32i l. (l33T l and (l37l i. But, we need to change the area where 
the function \/Np^jNp^Q^ x is integrated to the whole of real 
numbers. 

Theorem 11: When y|pj = v(q] 7 ^ e secon d order 
asymptotic expansions for a confidence coefficient < v < 1 
in (O are described as follows. 



L D (P,Q\v) = (H(P)/H(Q))n - ^/ 8V( ggf " + o{y/n). 
In particular, 



R 2 {PQ\v) = R»{P 1 Q\ V ) = 



>%V(P)\ogv-^ 



H{Q) 



(40) 



We note that Theorems [6] and [8] which we proved coincide 
with Theorem [4] and Theorem [2] in the limit Q — > U2 and 
P — > U2, respectively. Therefore, our results can be regard as 
extensions of existing studies in the limit. 

VI. Conclusion 

We treated the second order asymptotics of the random 
number generation from an i.i.d. probability distribution of 
P to that of Q. In existing studies, P or Q has been assumed 
to be a uniform distribution, but in this paper, both probability 
distributions have not been restricted to a uniform distribution. 
Let us review the proof of Theorem [6] in which we derived the 
limit of the maximal fidelity. Other theorems after Theorem |6] 
were variants of or derived from Theorem In the direct part, 
the maximal fidelity was achievable by performing different 
transformation to P™ on each area which was divided at points 
a and c. But we emphasize that d36i > is actually valid for 
any real numbers a and c. Therefore, the choice of a and c 
is not important in the direct part but essential in the direct 
part. In addition, the notion of the majorization was the key 
of the proof of the converse part since transformations in 
the sense of the majorization was wider than deterministic 



transformations, and thus the maximal fidelity under the ma- 
jorization condition gave an upper bound the maximal fidelity 
under deterministic transformations as (fj}. Finally, we remark 
that the majorization relation has an operational meaning as 
a transformation called LOCC for quantum entangled states 
in the quantum information theory, and our results can be 
extended to the quantum settings. A part of extensions of our 
results to quantum information theory is treated in |]9], H, 

GO- 
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